Lecture 1. Neural Networks¶

How to train your neurons

Dafnis Krasniqi (cours inspiré de Joaquin Vanschoren)

Overview¶

• Neural architectures
• Training neural nets
  • Forward pass: Tensor operations
  • Backward pass: Backpropagation
• Neural network design:
  • Activation functions
  • Weight initialization
  • Optimizers
• Neural networks in practice
• Model selection
  • Early stopping
  • Memorization capacity and information bottleneck
  • L1/L2 regularization
  • Dropout
  • Batch normalization

Linear models as a building block¶

• Logistic regression, drawn in a different, neuro-inspired, way
  • Linear model: inner product ($z$) of input vector $\mathbf{x}$ and weight vector $\mathbf{w}$, plus bias $w_0$
  • Logistic (or sigmoid) function maps the output to a probability in [0,1]
  • Uses log loss (cross-entropy) and gradient descent to learn the weights

Basic Architecture¶

• Add one (or more) hidden layers $h$ with $k$ nodes (or units, cells, neurons)
  • Every 'neuron' is a tiny function, the network is an arbitrarily complex function
  • Weights $w_{i,j}$ between node $i$ and node $j$ form a weight matrix $\mathbf{W}^{(l)}$ per layer $l$
• Every neuron weights the inputs $\mathbf{x}$ and passes it through a non-linear activation function
  • Activation functions ($f,g$) can be different per layer, output $\mathbf{a}$ is called activation $$\color{blue}{h(\mathbf{x})} = \color{blue}{\mathbf{a}} = f(\mathbf{z}) = f(\mathbf{W}^{(1)} \color{green}{\mathbf{x}}+\mathbf{w}^{(1)}_0) \quad \quad \color{red}{o(\mathbf{x})} = g(\mathbf{W}^{(2)} \color{blue}{\mathbf{a}}+\mathbf{w}^{(2)}_0)$$

More layers¶

• Add more layers, and more nodes per layer, to make the model more complex
  • For simplicity, we don't draw the biases (but remember that they are there)
• In dense (fully-connected) layers, every previous layer node is connected to all nodes
• The output layer can also have multiple nodes (e.g. 1 per class in multi-class classification)

Why layers?¶

• Each layer acts as a filter and learns a new representation of the data
  • Subsequent layers can learn iterative refinements
  • Easier that learning a complex relationship in one go
• Example: for image input, each layer yields new (filtered) images
  • Can learn multiple mappings at once: weight tensor $\mathit{W}$ yields activation tensor $\mathit{A}$
  • From low-level patterns (edges, end-points, ...) to combinations thereof
  • Each neuron 'lights up' if certain patterns occur in the input

Other architectures¶

• There exist MANY types of networks for many different tasks
• Convolutional nets for image data, Recurrent nets for sequential data,...
• Also used to learn representations (embeddings), generate new images, text,...

Training Neural Nets¶

• Design the architecture, choose activation functions (e.g. sigmoids)
• Choose a way to initialize the weights (e.g. random initialization)
• Choose a loss function (e.g. log loss) to measure how well the model fits training data
• Choose an optimizer (typically an SGD variant) to update the weights

Mini-batch Stochastic Gradient Descent (recap)¶

1. Draw a batch of batch_size training data $\mathbf{X}$ and $\mathbf{y}$
2. Forward pass : pass $\mathbf{X}$ though the network to yield predictions $\mathbf{\hat{y}}$
3. Compute the loss $\mathcal{L}$ (mismatch between $\mathbf{\hat{y}}$ and $\mathbf{y}$)
4. Backward pass : Compute the gradient of the loss with regard to every weight
   • Backpropagate the gradients through all the layers
5. Update $W$: $W_{(i+1)} = W_{(i)} - \frac{\partial L(x, W_{(i)})}{\partial W} * \eta$

Repeat until n passes (epochs) are made through the entire training set

Forward pass¶

• We can naturally represent the data as tensors
  • Numerical n-dimensional array (with n axes)
  • 2D tensor: matrix (samples, features)
  • 3D tensor: time series (samples, timesteps, features)
  • 4D tensor: color images (samples, height, width, channels)
  • 5D tensor: video (samples, frames, height, width, channels)

Tensor operations¶

• The operations that the network performs on the data can be reduced to a series of tensor operations
  • These are also much easier to run on GPUs
• A dense layer with sigmoid activation, input tensor $\mathbf{X}$, weight tensor $\mathbf{W}$, bias $\mathbf{b}$:

y = sigmoid(np.dot(X, W) + b)

• Tensor dot product for 2D inputs ($a$ samples, $b$ features, $c$ hidden nodes)

Element-wise operations¶

• Activation functions and addition are element-wise operations:

def sigmoid(x):
  return 1/(1 + np.exp(-x)) 

def add(x, y):
  return x + y

• Note: if y has a lower dimension than x, it will be broadcasted: axes are added to match the dimensionality, and y is repeated along the new axes

>>> np.array([[1,2],[3,4]]) + np.array([10,20])
array([[11, 22],
       [13, 24]])

Backward pass (backpropagation)¶

• For last layer, compute gradient of the loss function $\mathcal{L}$ w.r.t all weights of layer $l$

• Sum up the gradients for all $\mathbf{x}_j$ in minibatch: $\sum_{j} \frac{\partial \mathcal{L}(\mathbf{x}_j,y_j)}{\partial W^{(l)}}$
• Update all weights in a layer at once (with learning rate $\eta$): $W_{(i+1)}^{(l)} = W_{(i)}^{(l)} - \eta \sum_{j} \frac{\partial \mathcal{L}(\mathbf{x}_j,y_j)}{\partial W_{(i)}^{(l)}}$
• Repeat for next layer, iterating backwards (most efficient, avoids redundant calculations)

Backpropagation (example)¶

• Imagine feeding a single data point, output is $\hat{y} = g(z) = g(w_0 + w_1 * a_1 + w_2 * a_2 +... + w_p * a_p)$
• Decrease loss by updating weights:
  • Update the weights of last layer to maximize improvement: $w_{i,(new)} = w_{i} - \frac{\partial \mathcal{L}}{\partial w_i} * \eta$
  • To compute gradient $\frac{\partial \mathcal{L}}{\partial w_i}$ we need the chain rule: $f(g(x)) = f'(g(x)) * g'(x)$ $$\frac{\partial \mathcal{L}}{\partial w_i} = \color{red}{\frac{\partial \mathcal{L}}{\partial g}} \color{blue}{\frac{\partial \mathcal{g}}{\partial z_0}} \color{green}{\frac{\partial \mathcal{z_0}}{\partial w_i}}$$
• E.g., with $\mathcal{L} = \frac{1}{2}(y-\hat{y})^2$ and sigmoid $\sigma$: $\frac{\partial \mathcal{L}}{\partial w_i} = \color{red}{(y - \hat{y})} * \color{blue}{\sigma'(z_0)} * \color{green}{a_i}$

Backpropagation (2)¶

• Another way to decrease the loss $\mathcal{L}$ is to update the activations $a_i$
  • To update $a_i = f(z_i)$, we need to update the weights of the previous layer
  • We want to nudge $a_i$ in the right direction by updating $w_{i,j}$: $$\frac{\partial \mathcal{L}}{\partial w_{i,j}} = \frac{\partial \mathcal{L}}{\partial a_i} \frac{\partial a_i}{\partial z_i} \frac{\partial \mathcal{z_i}}{\partial w_{i,j}} = \left( \frac{\partial \mathcal{L}}{\partial g} \frac{\partial \mathcal{g}}{\partial z_0} \frac{\partial \mathcal{z_0}}{\partial a_i} \right) \frac{\partial a_i}{\partial z_i} \frac{\partial \mathcal{z_i}}{\partial w_{i,j}}$$
  • We know $\frac{\partial \mathcal{L}}{\partial g}$ and $\frac{\partial \mathcal{g}}{\partial z_0}$ from the previous step, $\frac{\partial \mathcal{z_0}}{\partial a_i} = w_i$, $\frac{\partial a_i}{\partial z_i} = f'$ and $\frac{\partial \mathcal{z_i}}{\partial w_{i,j}} = x_j$

Backpropagation (3)¶

• With multiple output nodes, $\mathcal{L}$ is the sum of all per-output (per-class) losses
  • $\frac{\partial \mathcal{L}}{\partial a_i}$ is sum of the gradients for every output
• Per layer, sum up gradients for every point $\mathbf{x}$ in the batch: $\sum_{j} \frac{\partial \mathcal{L}(\mathbf{x}_j,y_j)}{\partial W}$
• Update all weights of every layer $l$
  • $W_{(i+1)}^{(l)} = W_{(i)}^{(l)} - \eta \sum_{j} \frac{\partial \mathcal{L}(\mathbf{x}_j,y_j)}{\partial W_{(i)}^{(l)}}$
• Repeat with a new batch of data until loss converges
• Nice animation of the entire process

Backpropagation (summary)¶

• The network output $a_o$ is defined by the weights $W^{(o)}$ and biases $\mathbf{b}^{(o)}$ of the output layer, and
• The activations of a hidden layer $h_1$ with activation function $a_{h_1}$, weights $W^{(1)}$ and biases $\mathbf{b^{(1)}}$:

• Minimize the loss by SGD. For layer $l$, compute $\frac{\partial \mathcal{L}(a_o(x))}{\partial W_l}$ and $\frac{\partial \mathcal{L}(a_o(x))}{\partial b_{l,i}}$ using the chain rule
• Decomposes into gradient of layer above, gradient of activation function, gradient of layer input:

Activation functions for hidden layers¶

• Sigmoid: $f(z) = \frac{1}{1+e^{-z}}$
• Tanh: $f(z) = \frac{2}{1+e^{-2z}} - 1$
  • Activations around 0 are better for gradient descent convergence
• Rectified Linear (ReLU): $f(z) = max(0,z)$
  • Less smooth, but much faster (note: not differentiable at 0)
• Leaky ReLU: $f(z) = \begin{cases} 0.01z & z<0 \\ z & otherwise \end{cases}$

Effect of activation functions on the gradient¶

• During gradient descent, the gradient depends on the activation function $a_{h}$: $\frac{\partial \mathcal{L}(a_o)}{\partial W^{(l)}} = \color{red}{\frac{\partial \mathcal{L}(a_o)}{\partial a_{h_l}}} \color{blue}{\frac{\partial a_{h_l}}{\partial z_{h_l}}} \color{green}{\frac{\partial z_{h_l}}{\partial W^{(l)}}}$
• If derivative of the activation function $\color{blue}{\frac{\partial a_{h_l}}{\partial z_{h_l}}}$ is 0, the weights $w_i$ are not updated
  • Moreover, the gradients of previous layers will be reduced (vanishing gradient)
• sigmoid, tanh: gradient is very small for large inputs: slow updates
• With ReLU, $\color{blue}{\frac{\partial a_{h_l}}{\partial z_{h_l}}} = 1$ if $z>0$, hence better against vanishing gradients
  • Problem: for very negative inputs, the gradient is 0 and may never recover (dying ReLU)
  • Leaky ReLU has a small (0.01) gradient there to allow recovery

ReLU vs Tanh¶

• What is the effect of using non-smooth activation functions?
  • ReLU produces piecewise-linear boundaries, but allows deeper networks
  • Tanh produces smoother decision boundaries, but is slower

Activation functions for output layer¶

• sigmoid converts output to probability in [0,1]
  • For binary classification
• softmax converts all outputs (aka 'logits') to probabilities that sum up to 1
  • For multi-class classification ($k$ classes)
  • Can cause over-confident models. If so, smooth the labels: $y_{smooth} = (1-\alpha)y + \frac{\alpha}{k}$ $$\text{softmax}(\mathbf{x},i) = \frac{e^{x_i}}{\sum_{j=1}^k e^{x_j}}$$
• For regression, don't use any activation function, let the model learn the exact target

Weight initialization¶

• Initializing weights to 0 is bad: all gradients in layer will be identical (symmetry)
• Too small random weights shrink activations to 0 along the layers (vanishing gradient)
• Too large random weights multiply along layers (exploding gradient, zig-zagging)
• Ideal: small random weights + variance of input and output gradients remains the same
  • Glorot/Xavier initialization (for tanh): randomly sample from $N(0,\sigma), \sigma = \sqrt{\frac{2}{\text{fan_in + fan_out}}}$
    • fan_in: number of input units, fan_out: number of output units
  • He initialization (for ReLU): randomly sample from $N(0,\sigma), \sigma = \sqrt{\frac{2}{\text{fan_in}}}$
  • Uniform sampling (instead of $N(0,\sigma)$) for deeper networks (w.r.t. vanishing gradients)

Weight initialization: transfer learning¶

• Instead of starting from scratch, start from weights previously learned from similar tasks
  • This is, to a big extent, how humans learn so fast
• Transfer learning: learn weights on task T, transfer them to new network
  • Weights can be frozen, or finetuned to the new data
• Only works if the previous task is 'similar' enough
  • Meta-learning: learn a good initialization across many related tasks

Optimizers¶

SGD with learning rate schedules¶

• Using a constant learning $\eta$ rate for weight updates $\mathbf{w}_{(s+1)} = \mathbf{w}_s-\eta\nabla \mathcal{L}(\mathbf{w}_s)$ is not ideal
• Learning rate decay/annealing with decay rate $k$
  • E.g. exponential ($\eta_{s+1} = \eta_{s} e^{-ks}$), inverse-time ($\eta_{s+1} = \frac{\eta_{0}}{1+ks}$),...
• Cyclical learning rates
  • Change from small to large: hopefully in 'good' region long enough before diverging
  • Warm restarts: aggressive decay + reset to initial learning rate

Momentum¶

• Imagine a ball rolling downhill: accumulates momentum, doesn't exactly follow steepest descent
  • Reduces oscillation, follows larger (consistent) gradient of the loss surface
• Adds a velocity vector $\mathbf{v}$ with momentum $\gamma$ (e.g. 0.9, or increase from $\gamma=0.5$ to $\gamma=0.99$) $$\mathbf{w}_{(s+1)} = \mathbf{w}_{(s)} + \mathbf{v}_{(s)} \qquad \text{with} \qquad \color{blue}{\mathbf{v}_{(s)}} = \color{green}{\gamma \mathbf{v}_{(s-1)}} - \color{red}{\eta \nabla \mathcal{L}(\mathbf{w}_{(s)})}$$
• Nesterov momentum: Look where momentum step would bring you, compute gradient there
  • Responds faster (and reduces momentum) when the gradient changes $$\color{blue}{\mathbf{v}_{(s)}} = \color{green}{\gamma \mathbf{v}_{(s-1)}} - \color{red}{\eta \nabla \mathcal{L}(\mathbf{w}_{(s)} + \gamma \mathbf{v}_{(s-1)})}$$